Automatic Differentiation Research Articles

An advanced fast method for the evaluation of multiple immunolabelling using gold nanoparticles based on low-energy STEM

We present a powerful method for the simultaneous detection of Au nanoparticles located on both sides of ultrathin sections. The method employs a high-resolution scanning electron microscope (HRSEM) operating in scanning transmission electron microscopy (STEM) mode in combination with the detection of backscattered electrons (BSE). The images are recorded simultaneously during STEM and BSE imaging at the precisely selected accelerating voltage. Under proper imaging conditions, the positions of Au nanoparticles on the top or bottom sides can be clearly differentiated, hence showing this method to be suitable for multiple immunolabelling using Au nanoparticles (NPs) as markers. The difference between the upper and lower Au NPs is so large that it is possible to apply common software tools (such as ImageJ) to enable their automatic differentiation. The effects of the section thickness, detector settings and accelerating voltage on the resulting image are shown. Our experimental results correspond to the results modelled in silico by Monte Carlo (MC) simulations.

Discovering optimal kinetic pathways for self-assembly using automatic differentiation

Macromolecular complexes are often composed of diverse subunits. The self-assembly of these subunits is inherently nonequilibrium and must avoid kinetic traps to achieve high yield over feasible timescales. We show how the kinetics of self-assembly benefits from diversity in subunits because it generates an expansive parameter space that naturally improves the "expressivity" of self-assembly, much like a deeper neural network. By using automatic differentiation algorithms commonly used in deep learning, we searched the parameter spaces of mass-action kinetic models to identify classes of kinetic protocols that mimic biological solutions for productive self-assembly. Our results reveal how high-yield complexes that easily become kinetically trapped in incomplete intermediates can instead be steered by internal design of rate-constants or external and active control of subunits to efficiently assemble. Internal design of a hierarchy of subunit binding rates generates self-assembly that can robustly avoid kinetic traps for all concentrations and energetics, but it places strict constraints on selection of relative rates. External control via subunit titration is more versatile, avoiding kinetic traps for any system without requiring molecular engineering of binding rates, albeit less efficiently and robustly. We derive theoretical expressions for the timescales of kinetic traps, and we demonstrate our optimization method applies not just for design but inference, extracting intersubunit binding rates from observations of yield-vs.-time for a heterotetramer. Overall, we identify optimal kinetic protocols for self-assembly as a powerful mechanism to achieve efficient and high-yield assembly in synthetic systems whether robustness or ease of "designability" is preferred.

NG-DPSM: A neural green-distributed point source method for modelling ultrasonic field emission near fluid-solid interface using physics informed neural network

Distributed point source method (DPSM) is a collocation point-based semi-analytical method to model the scattered ultrasonic fields in isotropic and anisotropic materials. It requires the evaluation of Green's function solutions and its rigorous differentiation for complex geometry problems having a large set of distributed point sources. DPSM is much faster than the finite element method (FEM) however, it is analytically demanding as it requires differentiation of the Green's function. The intricacy associated with differentiation operation hinders the potential application of DPSM in large-scale automation and real-time structural health monitoring. The current paper introduces machine-learning-based strategies within DPSM and yields the proposed Neural Green-DPSM (NG-DPSM) framework. A physics-informed neural network (PINN) is incorporated in the DPSM framework for evaluating the Green's function and its gradients. To train the PINN, displacement Green's function solutions of wave propagation in isotropic solid is used as the governing physics. Once the PINN is trained for displacement Green's function solutions, the NG-DPSM framework leverages the trained network and its automatic differentiation capability to predict displacement and stress fields annihilating the rigorous differentiation of Green's functions. The accuracy and efficacy of NG-DPSM are demonstrated using two numerical experiments. First, the ultrasonic fields are evaluated for the problem geometry of a plate immersed in fluid excited at different angles of incidence. Further, the efficacy of the proposed approach is demonstrated for wave scattering through a circular hole in the plate. The results show a strong agreement between the ultrasonic fields computed using both NG-DPSM and conventional DPSM.

I❤MESH: A DSL for Mesh Processing

I❤MESH: A DSL for Mesh Processing

Exploring Physics-Informed Neural Networks for the Generalized Nonlinear Sine-Gordon Equation

The nonlinear sine-Gordon equation is a prevalent feature in numerous scientific and engineering problems. In this paper, we propose a machine learning-based approach, physics-informed neural networks (PINNs), to investigate and explore the solution of the generalized non-linear sine-Gordon equation, encompassing Dirichlet and Neumann boundary conditions. To incorporate physical information for the sine-Gordon equation, a multiobjective loss function has been defined consisting of the residual of governing partial differential equation (PDE), initial conditions, and various boundary conditions. Using multiple densely connected independent artificial neural networks (ANNs), called feedforward deep neural networks designed to handle partial differential equations, PINNs have been trained through automatic differentiation to minimize a loss function that incorporates the given PDE that governs the physical laws of phenomena. To illustrate the effectiveness, validity, and practical implications of our proposed approach, two computational examples from the nonlinear sine-Gordon are presented. We have developed a PINN algorithm and implemented it using Python software. Various experiments were conducted to determine an optimal neural architecture. The network training was employed by using the current state-of-the-art optimization methods in machine learning known as Adam and L-BFGS-B minimization techniques. Additionally, the solutions from the proposed method are compared with the established analytical solutions found in the literature. The findings show that the proposed method is a computational machine learning approach that is accurate and efficient for solving nonlinear sine-Gordon equations with a variety of boundary conditions as well as any complex nonlinear physical problems across multiple disciplines.

Multimodal surface wave inversion with automatic differentiation

Summary Investigating subsurface shear-wave velocity (vs) structures using surface wave dispersion data involves minimizing a misfit function that is commonly solved through gradient-based optimization. Sensitivity kernels for model updates are commonly estimated using numerical differentiation, variational methods or implicit functions which however, may involve numerical instability and computational challenges when dealing with complex velocity models and large datasets. In this study, we propose a novel surface wave inversion framework in which error-free gradients are calculated by automatic differentiation (AD) and forward modeling is implemented by convenient computational graphs in the state-of-the-art deep learning framework. The AD-based inversion approach is first validated using two synthetic datasets. Then the subsurface structures at three distinct locations, namely the Great Plains and the Long Beach in the US and Tong Zhou in China, are also derived using this method with seismic ambient noise data, which show nice consistency with those obtained using traditional methods. With the significantly improved computational efficiency, a great number of initial models can be inverted simultaneously to mitigate the impact of local minima and to estimate the uncertainty in the invert models. We have developed a new surface wave inversion package named ADsurf based on automatic differentiation and computational graphs in the deep learning framework, and its computational efficiency is also compared with the traditional finite-difference based gradient estimation approach. While a great number of intriguing studies on the geophysical inverse problems have been conducted recently using deep learning for end-to-end mapping, the use of AD provided in the in the deep learning frameworks to assist and expedite the gradient computations are still underexploited in geophysics. Thus, it is expected that various geophysical inverse problems in many different areas beyond the surface wave inversion can also be tackled with this new paradigm in the future.

Uncertainty-biased molecular dynamics for learning uniformly accurate interatomic potentials

Efficiently creating a concise but comprehensive data set for training machine-learned interatomic potentials (MLIPs) is an under-explored problem. Active learning, which uses biased or unbiased molecular dynamics (MD) to generate candidate pools, aims to address this objective. Existing biased and unbiased MD-simulation methods, however, are prone to miss either rare events or extrapolative regions—areas of the configurational space where unreliable predictions are made. This work demonstrates that MD, when biased by the MLIP’s energy uncertainty, simultaneously captures extrapolative regions and rare events, which is crucial for developing uniformly accurate MLIPs. Furthermore, exploiting automatic differentiation, we enhance bias-forces-driven MD with the concept of bias stress. We employ calibrated gradient-based uncertainties to yield MLIPs with similar or, sometimes, better accuracy than ensemble-based methods at a lower computational cost. Finally, we apply uncertainty-biased MD to alanine dipeptide and MIL-53(Al), generating MLIPs that represent both configurational spaces more accurately than models trained with conventional MD.

Robust and efficient implementation of finite strain generalized continuum models for material failure: Analytical, numerical, and automatic differentiation with hyper-dual numbers

Generalized continuum models for representing nonlinear material behavior including material failure in the finite strain regime are commonly formulated based on scalar elastic and dissipation potential functions. The evolution of stresses and internal variables, i.e., the material state, is governed by partial derivatives of the potential functions with respect to deformation and stress measures. Furthermore, for application of such models in implicit nonlinear finite element analysis tangent operators, consistent with the numerical integration algorithm, are required. In this work, we study analytical, numerical and automatic differentiation schemes for the implementation of coupled, generalized continuum models considering finite inelastic deformations and fracture. This includes the application of finite difference approximations, complex-step derivative approximations and automatic differentiation based on hyper-dual numbers. For the use of automatic differentiation, a semi-analytical split approach is introduced for increasing the computational efficiency. Based on a comprehensive 2D and 3D finite element study, we demonstrate the superior properties of automatic differentiation compared to numerical differentiation methods with regards to accuracy and robustness. Furthermore, the additional computational cost for automatic differentiation using the proposed split approach becomes negligible for large problems.

BEM-based fast frequency sweep for acoustic scattering by periodic slab

This paper presents a boundary element method (BEM) for computing the energy transmittance of a singly-periodic grating in 2D for a wide frequency band, which is of engineering interest in various fields with possible applications to acoustic metamaterial design. The proposed method is based on the Padé approximants of the response. The high-order frequency derivatives of the sound pressure necessary to evaluate the approximants are evaluated by a novel fast BEM accelerated by the fast-multipole and hierarchical matrix methods combined with the automatic differentiation. The target frequency band is divided adaptively, and the Padé approximation is used in each subband so as to accurately estimate the transmittance for a wide frequency range. Through some numerical examples, we confirm that the proposed method can efficiently and accurately give the transmittance even when some anomalies and stopband exist in the target band.

Geometrically exact beam theory for gradient-based optimization

Decades of research have progressed geometrically exact beam theory to the point where it is now an invaluable resource for analyzing and modeling highly flexible slender structures. Large-scale optimization using geometrically exact beam theory remains nontrivial, however, due to the inability of gradient-free optimizers to handle large numbers of design variables in a computationally efficient manner and the difficulties associated with obtaining smooth, accurate, and efficiently calculated design sensitivities for gradient-based optimization. To overcome these challenges, this paper presents a finite-element implementation of geometrically exact beam theory which has been developed specifically for gradient-based optimization. A key feature of this implementation of geometrically exact beam theory is its compatibility with forward and reverse-mode automatic differentiation. Another key feature is its support for both continuous and discrete adjoint sensitivity analysis. Other features are also presented which build upon previous implementations of geometrically exact beam theory, including a singularity-free rotation parameterization based on Wiener-Milenković parameters, an implementation of stiffness-proportional structural damping using a discretized form of the compatibility equations, and a reformulation of the equations of motion for geometrically exact beam theory as a semi-explicit system. Several examples are presented which verify the utility and validity of each of these features.

End-to-End Differentiable Physics Temperature Estimation for Permanent Magnet Synchronous Motor

Differentiable physics is an approach that effectively combines physical models with deep learning, providing valuable information about physical systems during the training process of neural networks. This integration enhances the generalization ability and ensures better consistency with physical principles. In this work, we propose a framework for estimating the temperature of a permanent magnet synchronous motor by combining neural networks with the differentiable physical thermal model, as well as utilizing the simulation results. In detail, we first implement a differentiable thermal model based on a lumped parameter thermal network within an automatic differentiation framework. Subsequently, we add a neural network to predict thermal resistances, capacitances, and losses in real time and utilize the thermal parameters’ optimized empirical values as the initial output values of the network to improve the accuracy and robustness of the final temperature estimation. We validate the conceivable advantages of the proposed method through extensive experiments based on both synthetic data and real-world data and then provide some further potential applications.

A Novel Algorithm for Solving High-Dimensional Poisson Equations Based on Radial Basis Function Neural Networks

As a widely used equation in electrostatics, the Poisson equation has significant research value in numerical solution. The basic principle of existing methods is to divide the solution domain into various grids and solve the numerical solutions at each grid node. Therefore, the accuracy of the solution is strongly correlated with the grid density divided. Based on this, this paper proposes a grid-free numerical calculation method that requires far fewer model parameters than traditional methods, and can ignore the order of the equation to solve high-dimensional Poisson equations. Given a Poisson equation, which has a certain type of boundary condition. A certain number of coordinate points are selected on the solution space and its boundary to construct a dataset. Using automatic differentiation technique to fit the differential operator in the equation, a loss function is constructed by incorporating the given boundary conditions or initial conditions, and the final numerical solution is obtained through iterative optimization algorithms. In the numerical experiment section, the algorithm proposed in this paper was used to solve the two-dimensional and three-dimensional Poisson equations with given exact solutions. The relative errors between the numerical solution and the true solution were [Formula: see text] and [Formula: see text], which are within the acceptable range. This proves that the proposed algorithm is feasible for solving the two-dimensional and three-dimensional Poisson equations with precise solutions. Secondly, the proposed algorithm is used to solve the four-dimensional Poisson equation with first-type boundary conditions, and the relative error range of the solution was within [0,0.56], which successfully extends the algorithm to solve high-dimensional Poisson equations and verifies its feasibility and efficiency in solving high-dimensional Poisson equations regardless of the dimension restriction.

Efficient excitation-transfer across fully connected networks via local-energy optimization

We study the excitation transfer across a fully connected quantum network whose sites energies can be artificially designed. Starting from a simplified model of a broadly-studied physical system, we systematically optimize its local energies to achieve high excitation transfer for various environmental conditions, using an adaptive Gradient Descent technique and Automatic Differentiation. We show that almost perfect transfer can be achieved with and without local dephasing, provided that the dephasing rates are not too large. We investigate our solutions in terms of resilience against variations in either the network connection strengths, or size, as well as coherence losses. We highlight the different features of a dephasing-free and dephasing-driven transfer. Our work gives further insight into the interplay between coherence and dephasing effects in excitation-transfer phenomena across fully connected quantum networks. In turn, this will help designing optimal transfer in artificial open networks through the simple manipulation of local energies.

Simulation of a spin-boson model by iterative optimization of a parametrized quantum circuit

Time evolution of the populations of spin states coupled with bosons, which can be a model of photosynthetic excitation energy transfer of dye molecules surrounded by proteins, is simulated using the projected-variational quantum dynamics algorithm. By a transformation of the Hamiltonian describing the spin-boson model into the one-dimensional nearest-neighbor form, it is shown that the spin-boson model can be simulated using the sequential ansatz even if a quantum computer has limited connectivity. The optimization of the parametrized quantum circuits is performed by the gradient descent method on a classical computer using the automatic differentiation, and the population of the spins is simulated on a noisy intermediate-scale quantum computer. The error originating from the quantum computing is mitigated by the Clifford data regression, in which the noise channel is estimated using the data obtained from all the time steps.

Discovery of magnesium-aluminum alloys by generative model and automatic differentiation approach

Abstract Magnesium-aluminum alloy is one of the most common alloy materials in the industry, widely utilized due to its low density and excellent mechanical properties. However, investigating its properties or predicting new structures through experiment inevitably involves complex processes, which cost much time and money. Currently, simulations, such as density functional theory (DFT) and machine learning (ML) methods, are mainly employed for predicting new alloy materials. While precise, DFT incurs significant computational costs, which posing challenges for research. On the other hand, although ML methods are versatile and efficient, they demand high-quality datasets and may exhibit some degree of inaccuracy. To address these challenges, we employ a combination of generative model and automatic differentiation, reducing the search space and accelerating the screening of target materials. Finally, we use generative model to predict a multitude of magnesium-aluminum alloys. We perform structure optimization and property evaluation for ten potentially valuable intermetallic compounds. Ultimately, we identified Mg3Al3, Mg2Al6, Mg4Al12, Mg15Al, and Mg14Al2 as five stable structures, among which Mg4Al12, Mg15Al and Mg14Al2 may have higher potential application value.

VIPurPCA: Visualizing and Propagating Uncertainty in Principal Component Analysis.

Variables obtained by experimental measurements or statistical inference typically carry uncertainties. When an algorithm uses such quantities as input variables, this uncertainty should propagate to the algorithm's output. Concretely, we consider the classic notion of principal component analysis (PCA): If it is applied to a finite data matrix containing imperfect (i.e., uncertain) multidimensional measurements, its output-a lower-dimensional representation-is itself subject to uncertainty. We demonstrate that this uncertainty can be approximated by appropriate linearization of the algorithm's nonlinear functionality, using automatic differentiation. By itself, however, this structured, uncertain output is difficult to interpret for users. We provide an animation method that effectively visualizes the uncertainty of the lower dimensional map. Implemented as an open-source software package, it allows researchers to assess the reliability of PCA embeddings.

An inverse analysis of fluid flow through granular media using differentiable lattice Boltzmann method

This study presents a method for the inverse analysis of fluid flow problems. The focus is put on accurately determining boundary conditions and characterizing the physical properties of granular media, such as permeability, and fluid components, like viscosity. The primary aim is to deduce either constant pressure head or pressure profiles, given the known velocity field at a steady-state flow through a conduit containing obstacles, including walls, spheres, and grains. The lattice Boltzmann method (LBM) combined with automatic differentiation (AD) (AD-LBM) is employed, with the help of the GPU-capable Taichi programming language. A lightweight tape is used to generate gradients for the entire LBM simulation, enabling end-to-end backpropagation. Our AD-LBM approach accurately estimates the boundary conditions for complex flow paths in porous media, leading to observed steady-state velocity fields and deriving macro-scale permeability and fluid viscosity. The method demonstrates significant advantages in terms of prediction accuracy and computational efficiency, making it a powerful tool for solving inverse fluid flow problems in various applications.

Enhancing predictive capabilities in data-driven dynamical modeling with automatic differentiation: Koopman and neural ODE approaches.

Data-driven approximations of the Koopman operator are promising for predicting the time evolution of systems characterized by complex dynamics. Among these methods, the approach known as extended dynamic mode decomposition with dictionary learning (EDMD-DL) has garnered significant attention. Here, we present a modification of EDMD-DL that concurrently determines both the dictionary of observables and the corresponding approximation of the Koopman operator. This innovation leverages automatic differentiation to facilitate gradient descent computations through the pseudoinverse. We also address the performance of several alternative methodologies. We assess a "pure" Koopman approach, which involves the direct time-integration of a linear, high-dimensional system governing the dynamics within the space of observables. Additionally, we explore a modified approach where the system alternates between spaces of states and observables at each time step-this approach no longer satisfies the linearity of the true Koopman operator representation. For further comparisons, we also apply a state-space approach (neural ordinary differential equations). We consider systems encompassing two- and three-dimensional ordinary differential equation systems featuring steady, oscillatory, and chaotic attractors, as well as partial differential equations exhibiting increasingly complex and intricate behaviors. Our framework significantly outperforms EDMD-DL. Furthermore, the state-space approach offers superior performance compared to the "pure" Koopman approach where the entire time evolution occurs in the space of observables. When the temporal evolution of the Koopman approach alternates between states and observables at each time step, however, its predictions become comparable to those of the state-space approach.

Controllable and scalable gradient-driven optimization design for two-dimensional metamaterials based on deep learning

A novel gradient-driven rapid optimization design method based on deep learning is proposed to design two-dimensional irregular elastic metamaterials (EMMs). We develop a design network comprising a conditional topology generator (TG) and a bandgap predictor (BGP). TG is applied to controllably generate candidate topological structures with geometric constraints, while BGP serves as a surrogate model linking EMM structures with their bandgaps. Utilizing automatic differentiation techniques, the error gradients of design variables are computed on the design network and then passed to gradient-driven optimization algorithms to optimize the design variables until the target bandgap is achieved. The design variables in optimization algorithms are independent of the design network’s input features, allowing adaptation to changes in design factors without retraining the network, thereby providing scalability. The testing results show that the proposed method can efficiently and rapidly design EMM structureswith significant consistency to target bandgaps. Additionally, further comparative analysis reveals that the irregular topological structures enable bandgap broadening, generation of new bandgaps, and band flattening. The proposed method demonstrates remarkable feasibility and efficiency in the design of irregular EMM structures, and offers practical controllability and scalability.

Taming Binarized Neural Networks and Mixed-Integer Programs

There has been a great deal of recent interest in binarized neural networks, especially because of their explainability. At the same time, automatic differentiation algorithms such as backpropagation fail for binarized neural networks, which limits their applicability. We show that binarized neural networks admit a tame representation by reformulating the problem of training binarized neural networks as a subadditive dual of a mixed-integer program, which we show to have nice properties. This makes it possible to use the framework of Bolte et al. for implicit differentiation, which offers the possibility for practical implementation of backpropagation in the context of binarized neural networks. This approach could also be used for a broader class of mixed-integer programs, beyond the training of binarized neural networks, as encountered in symbolic approaches to AI and beyond.